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Abstract 


In this paper the performance of Pattern storage 
feedback neural network is evaluated for the storage 
and recalling of finger prints using unsupervised 
leaming. The paper first discusses the storage and 
recall via hebbian leaming and the problem areas or 
the efficiency issues involved and then the 
performance enhancement via the pseudo-inverse 
learning. Performance is measured with respect to 
storage capacity, recall of distorted or noisy patterns 
i.e. association of a noisy version of a stored patter to 
the original stored pattern for testing the accretive 
behaviour of the network and association of new or 
noisy form of some memorized pattern. 
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Introduction 


on in the human brain refers to the 
phenomenon of one thought causing us to think 


of another. Correspondingly, associative 
memory is the function where the brain is able to 
store and recall information, given partial 
knowledge of the information content [1]. 
Associative Memory is a dynamic system which 
has a number of stable states with a domain of 
attraction around them. If the system starts at 
any state in the domain, it will converge to the 
locally stable state, which is called an attractor 
[2]. One such model, describing the 
organization of neurons in such a way that they 
function as Associative Memory is called 
Content Addressable Memory. The Neural 
Networks designed to model the process of 
memory recall in the human brain [30], were 
Proposed by J. J. Hopfield and was named after 
him as Hopfield Model. It is a fully connected 
neural network model in which patterns can be 
stored by distributing among neurons and we 
can retrieve one of the previously presented 
patterns from an example which is similar to, ora 
noisy version of it [1, 2]. The network associates 
each element of a pattern with a binary neuron. 
The neurons are updated asynchronously and 
in parallel. They are initialized with an input 
pattern and the network activation converges to 
the closest learnt pattern [18]. This dynamical 


behavior of the neurons strongly depends on the 
synaptic strength between neurons. The 
specification of the synaptic strength is 
conventionally referred to as learning [3]. 
Learning employs a number of learning 
algorithms as perceptron, hebbian, pseudo- 
inverse, LMS etc, [32]. While choosing a learning 
algorithm, there are a number of considerations. 
Those considered in this paper are: 


a) The maximum capacity ofthe network 


b) The network's ability to add patterns 
incrementally to the network 


c) The network's ability to correctly recall 
patterns stored in the network 


d) The network's ability to associate a noisy 
pattern to its original pattern 


e) The network's ability to associate a new 
pattern with its nearest neighbour 


Points d and e would also vary with the number of 
patterns currently stored in the network. 


The simplest rule that can be used to train a 
network is the hebbian rule. But it suffers from a 
number of problems such as: 


a) The maximum capacity Is limited to just 
0.14N, where N is the number of neurons in 
the network [32]. 


b) The recall efficiency of the network 
deteriorates as the number of patterns stored 
in the network increases [29, 33]. 


c) The network's ability to correct noisy patterns 
is also extremely limited and deteriorates with 
packing density of the networl. 


d) New patterns could hardly be associated to 
the stored patterns. 


The next rule to be considered to overcome the 
disadvantages of the hebbian rule was the 
pseudo-inverse learning rule. The standard 
pseudo-inverse rule is known to be better than 
the hebbian rule in terms of the capacity (N), 
recall efficiency and pattern corrections [30, 32]. 
This paper is organized with the above five 
questions in mind. Section Il provides a brief 
description of the Hopfield network as 
associative memory and its storage and update 


dynamics. Section Ill elaborates the Pseudo- 
inverse Rule, the associated problems and 
measures to overcome them. Section IV 
contains the experiments whose results have 
been compiled with in Section V. Discussions 
and conclusions then follow in section VI and VII 
respectively 


2. Hopfield Network as Associative Memory 


Hopfield Network is probably the best known 
example of a neural network working as 
associative memory [24, 32]. It is a fully 
connected network made up of bipolar threshold 
logic units. The units receive inputs from every 
other unit except for itself. The net input of a uniti 
at any time tis computed by 


s(t) = wy s(t) (1) 
jei 


where w, is the weight of the connection 
between unit i and j and s, is the state of unit j, 
which can be either +1 or-1 [12, 33]. 


The next state of unit i is a function of its net input 
and current state and is given by 


g(ttl) = {4 if s(t) >, (2) 
-1 if s(t) < 6, 
0; is assumed to be 0. 


The network's weight matrix W is an N x N 
matrix, whose contents are determined by the 
set of patterns and the learning rules used to set 
the weights. The set of stored patterns P={&1, €2, 
... §n) where each pattern €i is a vector of size n. 
Thus P is a matrix of size | x n, where | is the 
number of patterns stored in the network [32 — 
34]. The weight matrix in the current paper is a 
symmetric zero diagonal matrix. 


Pattern recall involves setting the initial state of 
the network equal to an input vector gi. The states 
of the individual units are then updated 
repeatedly until the overall state of the network is 
stable. Updation of units may be synchronous or 
asynchronous [1, 22]. In the synchronous 
update all the units of the network are updated 
simultaneously and the state of the network is 
frozen until update is made for all the units. While 
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in the asynchronous update, a unit is selected at 
random and its state is updated, using the 
current state of the network. This update via 
random choice of a unit is continued until no 
further change in the state takes place for all the 
units i.e. the network reaches a stable state. 
Each stable state of the network corresponds to 
a stored pattern that has a minimum hamming 
distance from the input pattern [11]. With each 
stable state of the network is associated an 
energy E and hence that state acts as a point 
attractor. During update the network moves from 
an initial high energy state to the nearest 
attractor. All stable states which are similar to any 
of €i of P are called Fundamental Memories. 
Apart from them there are other stable states, 
including inverses of fundamental memories, 
The number of such fundamental memories and 
the nature of additional stable states depend 
upon the learning algorithm that is employed. 


3. Storage of pattern 


The pseudo inverse method has been used for 
the storage of patterns in the form of images of 
finger prints. In the pseudo inverse, the weight 
matrix is given by: 

=Ea! (3) 
where = is the matrix whose rows are €n and=-1 


is its pseudo-inverse. The matrix with the 
property that =-1= = | [29, 32, 33]. 


The pseudo inverse rule is neither local nor 
incremental as compared to the hebbian rule, 
This means that the update of a particular 
connection does not depend on the information 
available on either side of the connection and 
also patterns cannot be incrementally added to 
the network. These problems can be solved by 
modifying the rule in such a way that some 
characteristics of hebbian learning are also 
incorporated so that locality and increment are 
ensured. The hebbian rule is given as: 

Wij = 1/N % fi Sy for i+j (4) 

= 0, for i=j, 1<i, j<N 

where, N is the number of units/neurons in the 
network. 


Siifor | = 1 to L are the vectors / images to be 
stored, where each component of €1 is binary i.e. 


each§\= +1 forj=1toN. 


Now the pseudo inverse of the weight matrix can 
be calculated as 


Woinv = W'* CW * wy! (5) 
where 
Wtis the transpose of the weight matrix W 


(w * w'y' is the inverse of the product of W and 
its transpose, 


This method will overcome the locality and 
increment problems associated with the 
pseudo-inverse rule. In addition, it has the 
benefits of the pseudo-inverse rule in terms of 
the storage capacity and recall efficiency over 
the hebbian rule. 


4. Implementation 


Various experiments were conducted to test the 
efficiency of the pseudo-inverse rule for the 
points mentioned in the Section. 


1. All experiments were conducted in Matlab. 
Before implementing the learning rule the initial 
task was conversion of raw fingerprint images 
into patterns for storage in the network. For this 
the images were preprocessed through a series 
of steps, briefly discussed in the following 
subsection. 


4.1, Image Preprocessing 


Preprocessing, in the form of image 
enhancement, of the fingerprint images is 
required to convert the images into suitable 
patterns for storage in the Hopfield Network. The 
term image enhancement refers to making the 
image clearer for easy further operations, The 
fingerprint images considered for the study are 
the images of the fingerprint impressions of 
different individuals. The images are not of 
perfect quality to be considered for storage in a 
network. Hence enhancement methods are 
required to reveal the fine details of the images 
which may remain uncovered due to insufficient 
ink or imperfect impressions. The enhancement 
methods would increase the contrast between 
image components and connect the broken or 
incomplete image components. 


The images were first scanned as RGB images _ 
and then converted to grayscale to retain the fine 


details in the images. Then the image was 
enhanced and made clearer and sharper using 
histogram equalization techniques. Note that 
histogram equalization refers to expansion of the 
pixel value distribution of an image so as to 
increase the perceptional information. The 
image was then subjected to binarization, 
Binarization refers to conversion of a gray scale 
image to a black and white image. Typically 
binarization converts an image of up to 256 gray 
levels to a black and white image. 


(b) (c) (a) 


Fig 1; (a) Initial RGB Image, (b) Gray scale Image, 
(c) Histogram equalized Image (d) Binarized 
Image 


After attaining binarization, the need was to 
convert the binary image to a bipolar image, 
since Hopfield networks work best with bipolar 
data. A bipolar image is one where each pixel has 
value either +1 or -1, Hence, all pixel values are 
verified and those with value 0 are converted to - 
1, thus converting the image to bipolar. Finally 
the image is converted to bipolar vectors. All the 
image vectors are stored into a comprehensive 
matrix which has the following structure: 


Sin fiz, 
P= Sau ba 
Saar aay veeeees GON 


4.2. Pattern Storage 


The patterns in the form of bipolar vectors 
created in section 4.1 were then stored in the 
Hopfield network via the following algorithm 
separately for hebbian and pseudo-inverse rules 
in separate weight matrices. 


Algorithm: Pattern Storage 
1.Begin with a zero weight matrix, W of size N x N 


2.Compute the weight matrix as per equation 4 
for the | patterns, where each pattern is a vector 
ofsizen. 


3.Assign zeros to the diagonal elements at the 
end of each iteration. 


4.Finally calculate the pseudo-inverse of W as 
per equation 5. 


Hence the L patterns are all stored in the Hopfield 
network. The final weight matrix is a symmetric 
zero diagonal matrix. Once the patterns were 
stored, the next experiments were to test the 
capacity and recall efficiency of the network. This 
is discussed in the next subsection. 


4.3, Pattern Recall 


4.3.1. Mathematical Implementation Pattern 
recall refers to the identification and retrieval of 
the corresponding image when an image is 
presented as input to the network, As soon as an 
image is fed as input to the network, the network 
starts updating itself. In the current paper, we use 
asynchronous update of the network units to find 
their new states. This update via random choice 
of a unit is continued until no further change in 
the state takes place for all the units. That is, the 
state at time (t+-1) is the same as the state at time 
tfor all the units. 


s(t+1)=s,(t) foralli (6) 


Such a state is referred to as the stable state. Ina 
stable state the output of the network will be a 
stable (trained) pattern that has a minimum 
hamming distance from the input pattern [11]. 
The network is said to have converged and 
recalled the pattern if the output matches the 
pattern presented initially as input. For pattern 
association, the patterns stored in an associative 
memory act as attractors and the largest 
hamming distance within which almost all states 
flow to the pattern is defined as the radius of the 
basin of attraction [5]. 


Each state of the Hopfield Network is associated 
with an energy value, whose value either 
reduces or remains the same as the state of the 
network changes [22], The energy function of 
the network is given by: 


V=-1/2 2 Twj 8, sj + 6s; (7) 
Tj 


Hopfield has shown that for symmetric weights 
with no self feedback connections and bipolar 
output functions, the dynamics of the network 
using asynchronous update always lead 
towards energy minima at equilibrium. The 
network states corresponding to these energy 


ai 
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minima are termed as stable states [22] and the 
network uses each of these stable states for 
storing individual patterns. 


The Hopfield network has the ability to recognize 
unclear pictures correctly. This means that the 
network can recall actual patterns when the 
noisy or partial clues of that pattern are 
presented to the network. It is known and has 
been shown [1, 2] that Hopfield networks 
converge to the original patterns if up to 40% 
distorted version ofa stored pattern is presented. 
The patterns are stored in the network in the form 
of attractors on the energy surface. The network 
can then be presented with either a portion of 
one of the images (partial cue) or an image 
degraded with noise (noisy cue) and through 
multiple iterations it will attempt to reconstruct 
one of the stored images. 


4.3.2 Algorithmic Impiementation: The 
algorithm for pattern recall in a Hopfield Neural 
Network storing L patterns is as follows: 


The algorithm would be implemented both for 
Hebbian and Pseudo-inverse rules and results 
would be recorded. Assume a pattern x, of size 
N, already stored in the network and modified by 
altering the values of k randomly chosen pixels. 
Also, assume vectors new to store the new states 
of the network. Consider a variable count 
initialized to value 1, 


1) Initialize weights to store patterns (use 
Hebbian and Pseudo-inverse Rule) as per 
algorithm for Pattern Storage, while 
activations of the net are converged perform 
steps 2to8. 


2) Foreach input vector x, repeat steps 3 to 7. 


3) Set initial activations of the net equal to the 
external input vector x, yi = xi (i=1 to n). 


4) Performsteps 5 to 7 for each unity, 
5) Compute the netinput 


Ynew = Hit ZB ys * wii 


6) Determine the activation (output signal) 


By = eh if Ynew > 0 
-1, if Ynew <= 0 
& 
7) Broadcast the value of ynew to all other 
units. 


8) Test for convergence as per equation 4. 


Note: The value of threshold 6 is assumed to be 
zero. Each unitis randomly chosen for update, 


The maximum number of patterns successfully 
recalled by the above procedure is a pointer to 
the maximum storage capacity of the Hopfield 
Network, which is further discussed in the 
results. Further the recall efficiency for noisy 
patterns is also determined as up to what 
percentage of error in the patterns is acceptable 
by the network and convergence to the original 
pattern occurs. This is discussed in the results. 


5. Results 


1) Experiment 1 (Pattern Recal ability and 
Critical Storage Capacity of the Hopfield 
Network). Table 1 shows the maximum 
number of patterns that can be efficiently 
stored in the Hopfield network with hebb 
and pseudo- inverse rule. 


2) Experiment 2 (Recall Efficiency of the 
Hopfield Network for Noisy Patterns and 
New Patterns). Table 2 compares for hebb 
and pseudo-inverse rule, the successful and 
unsuccessful recall results at 30%, 40%, 
50% and 60% distortion respectively at 
various packing densities of the network. 
The tables also show the results that when 
new patterns were presented to the 
network, whether they were associated to 
some stored pattern that matched them 
most closely or not. 


6. Discussions 
6.1. Pattern Storage 


All the patterns after preprocessing as depicted 
in Section 4.1 were converted into bipolar 
vectors ready for storage into the Network. As 
per the algorithm of 4.2, the patterns were Input 
one by one into the Hopfield Network first by 
hebbian rule and then by pseudo-inverse rule. 
The weight matrices for both are 900 x 900 zero 
diagonal, symmetric matrix. 


6.2. Pattern Recall and Critical Storage 
Capacity 

The storage capacity of a neural network refers 
to the maximum number of patterns that can be 
stored and successfully recalled for a given 
number of nodes, N. The Hopfield network is 
limited in storage capacity to 0.14N when trained 
with hebbian rule [1-5, 11-12, 18, 21]. But the 
capacity enhances to N with pseudo-inverse 


rule. Experiments were conducted to check the 
same and the network was able to store and 
perfectly recall 0.14N i.e. 126 patterns with 
hebbian rule and N i.e. 900 patterns with pseudo- 
inverse rule. 


Thus the critical storage capacity for the Hopfield 
Network comes out to be 0.14N with hebbian and 
N with pseudo-inverse rule without any error in 
pattern recall, for the current study. 


TABLE 1: CRITICAL STORAGE CAPACITY 


Hebbian Pseudo-inverse 
Capacity 126 900 


6.3. Recall of Noisy Patterns 


The Hopfield network has the ability to recognize 
unclear pictures correctly. This capacity of the 
network was tested as per algorithm in Section lV 
part C. This time the experiments were 
conducted with the network storing 20, 50, 80, 
100, 130 patterns successively. The patterns 
were modified by altering 30%, 40%, 50% and 
60% bits. The bits were randomly selected for 
alteration and the results compiled. Table 2 
displays the results for both rules at various 
packing densities of the network and various 
levels of distortion. Further the network's 
behavior is also tested for new patterns and the 
observations are recorded in the same table. 


Observations 


1) With 20 patterns, behaviour with distorted 
patterns is similar with both the rules i.e. up to 
40% distortion the same pattern is associated 
but at 50% distortion some other stored 
pattern or the nearest neighbour is associated, 
New patterns are not recognized by the 
hebbian rule but by pseudo-inverse, they are 
associated with some stored pattern. 


2)With 50 patterns, behaviour upto 40% 


TABLE 2: PATTERN RECALABILITY WITH NOISY 
AND NEW PATTERNS 


No. of Pattern Associated to 
Patterns | i 
in the 
Network | %distortion Hebbian Pseudoinverse 
30% Original Pattern Original Pattern 
40% Original Patten | Original Pattern 
20 Nearest 
patterns 50% Nearest neighbour neighbour 
60% None None 
New Nearest 


Pattems None Neighbour 
30% Original Pattern __| Original Pattern 
40% Original Patter _| Original Pattern | 
50 Nearest 
patterns 50% None Neighbour 
60% None None 
New ‘Nearest 
Patterns: None Neighbour 
Original for 50% 
30% pattems Original Pattem 
0 40% None Original Pattern 
patterns 50% None None 
60% None None 
New 
Patterns None None 
Original for 30% 
30% patterns Original Pattern 
40% None Original Pattern | 
patterns 50% None None 
60% None None 
New 
Patterns None None 
Original for 10% 
30% pattems: Original Pattern 
130 40%. None None 
patterns 50% None None 
60% None None 
New 
Patterns None ‘None 


distorted patterns is similar with both the rules. 
But at 50% distortion hebb rule stops the 
recognition of the pattern while pseudo- 
inverse associates it to its nearest neighbour. 
New patterns are not recognized by the 
hebbian rule but by pseudo inverse, they are 
associated to some stored pattern. 


3) With 80 patterns hebb rule further deteriorates 
and even at 30% distortion recognizes only 
50% patterns. The pseudo inverse rule on the 
other hand recognizes the patterns up to 40% 
distortion also but now it stops associating 
new patterns to stored patterns. 


4) With 100 patterns, hebb rule further 
deteriorates and recognizes only 30% 
patterns at 30% distortion. While pseudo 
inverse has the same behaviour as that with 80 
patterns. 


5) With 130 patterns, hebb rule recognizes 
hardly 10% patterns at 30% distortion. While 
pseudo inverse rule, though having degraded 
in performance at this stage yet is able to 
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recognize the original pattern up to 30% 
distortion. Beyond this there is no recognition, 


It was observed that the network performs 
sufficiently well with pseudo-inverse rule than 
with hebbian rule with regard to distorted 
patterns and with new patterns which are not 
at all recognized but the network in case of 
hebbian rule. 


Further it has been observed that the 
network's efficiency starts deteriorating as the 
network gets saturated, As depicted in table 2, 
the performance of the network deteriorated 
with 80 patterns for hebb rule and 130 patterns 
for pseudo inverse rule, This result can be 
attributed to the reduction of the Hamming 
Distance between the stored patterns and the 
consequent reduction of the radius of the 
basin of attraction of each stable state. Hence 
only few patterns could settle into the stable 
states of their original patterns. 


Conclusions 


The Hopfield network designed to work with 
image patterns works variably with the learning 
rules utilized to train the network. The efficiency 
is invariably low when we work with the hebbian 
rule in terms of the maximum capacity and the 
recall efficiency with noisy and new patterns. But 
this capability can be sufficiently enhanced when 
the pseudo inverse rule with capabilities of 
hebbian rule is used to train the network. With 
this, the capacity suddenly increases to N and 
the recall efficiency is also remarkably improved 
for distorted patterns. New patterns are now 
recognized by the network up to a limited 
packing density which was totally absent with 
hebb rule. 
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